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ABSTRACT 


What effects does PBL have on student learning outcomes, knowledge acquisition, and higher order thinking skills? This 
question continues to influence the adoption and implementation of problem-based learning (PBL) since its’ inception. 
The answer to this overarching question is still lacking. This chapter responds to this question by first analyzing and 
synthesizing the literature regarding the factors that influence the effectiveness of PBL to identify questions that 
designers and researchers should ask about results of learning. It then explores important characteristics of PBL 
compared with its’ analogous pedagogies and approaches of project-based learning, case-based reasoning, inquiry-based 
learning, learning design and/or design thinking in an effort to understand how features of each approach influence the 
effectiveness of learning outcomes. Further, the chapter examines how PBL approaches and pedagogies define learning 
outcomes and how various learning outcomes are measured. Finally, the chapter offers a synthesis of research on the 
effectiveness of PBL and recommendations for the future researchers and designers. 
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1. INTRODUCTION 


Problem-based learning (PBL) is an instructional approach rooted in constructivist and experiential learning 
theories. Research and theory on learning suggest that by having students learn through the experience of 
solving problems, they can learn both content and thinking strategies (Hmelo-Silver, 2004; Savery, 2006). In 
PBL, student learning centers around a complex problem that does not have a single correct answer or 
solution. Students learn content, strategies, and self-directed learning skills through collaboratively engaging 
in problem-solving, reflecting on their own experiences, and engaging in self-directed inquiry (Hmelo-Silver, 
Duncan, & Chinn, 2007). It has been maintained in the literature that PBL positively influences learning 
outcomes along with learners’ higher order thinking skills such as creative thinking, problem-solving, logical 
thinking and decision making (Senda& & Odabasi, 2009). With the advent of reform movements in 
education, such as 21" Century Learning Skills, PBL is increasingly being advocated for and adopted by 
institutions of higher education. 

However, while the utilization of PBL has expanded, disputes exist regarding the amount of scaffolding 
and guidance provided to students during implementation. Scaffolding is a temporary guidance provided to 
assist learners with the learning process. Kirschner, Sweller and Clark (2006) argue that PBL represents a 
minimally guided instructional practice. In contrast, Hmelo-Silver, Duncan and Chinn (2007) argue that PBL 
is not a minimally guided instructional approach, but “rather provides extensive scaffolding and guidance to 
facilitate student learning” (p. 99). 

This dispute is relevant in higher education as the rate of online education programs continue to rise. PBL 
has traditionally been conducted in face-to-face settings using cooperative learning groups. Less is known 
regarding the successful implementation and facilitation of PBL online. Online education has emerged as a 
popular alternative to face-to-face classroom instruction. Most online courses are delivered asynchronously, 
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allowing instruction and communication between students and instructors to occur independent of time and 
location. However, the traditional implementation of PBL poses significant challenges to asynchronous 
online instructors in terms of scaffolding both individuals and groups. 

The purposes of this study were to (1) design, develop and pilot test five self-directed, computer-based 
modules to support scaffolding in an online graduate level course utilizing Problem/Project-Based Learning 
(PBL), (2) collect data to assess the effectiveness of computer-supported scaffolding (hard scaffold) provided 
in the modules to assist students in problem identification, application of conceptual and domain-specific 
knowledge and skills of argumentation, and (3) use the results of the data to identify recommendations for 
future researchers and designers. 


2. REVIEW OF THE LITERATURE 


PBL is one of a family of constructivist, experiential learning approaches, which situate learning in a 
meaningful task (Hmelo-Silver, 2004), such as project-based learning (PrBL) and inquiry learning (IL) 
(Savery, 2006). Despite some differences among these approaches, the focus of each is a question, issue, case 
or problem that learners attempt to solve or resolve (Jonassen, 1999). In addition, problem-based approaches 
share the same assumptions about active, constructive and authentic learning experiences (Jonassen, 1999). 
Due to the similar characteristics, problem-based approaches such as PBL (Problem-based Learning), Pj]BL 
(Project-based Learning) and IL (Inquiry-based Learning) are often used in combination and play 
complementary roles in practice. 


2.1 Process of Implementing PBL 


PBL, as its name implies, situates learning in the context of a problem. The PBL learning cycle, also known 
as the PBL tutorial process, typically starts with the presentation of a problem rather than a lecture or reading 
assignment intended to impart discipline-specific knowledge to the student (Savery, 2009). The problem is 
ill-structured in nature and refers to an academically or professionally relevant issue (Yew & Schmidt, 2012) 
that students learn more about through the process of investigation and production of viable solutions. 
Students work in small collaborative groups to identify relevant facts from the provided problem scenario. As 
a group, students analyze the problem, generate possible explanations, as well as identify key issues and 
concepts they need to learn more about in order to solve the problem (Hmelo-Silver, 2004; Savery, 2009; 
Yew & Schmidt, 2012). After this period of teamwork, students disperse for a phase of self-directed study. 
Students independently research and investigate selected learning issues identified by the group. “They then 
regroup to share what they have learned, reconsider their hypotheses, and/or generate new ideas in light of 
their new learning” (Hmelo-Silver, 2004, p. 242). A tutor/facilitator is present during the group discussions to 
help facilitate the learning processes and the development of metacognitive skills (Savery, 2009; Yew 
& Schmidt, 2012). 

PBL tutors/facilitators do not directly transmit/teach the content knowledge to students. Instead they 
support the students’ learning process by observing the students, pushing them to think deeply by asking 
probing questions and encouraging students to articulate their thinking, modeling problem-solving strategies, 
and promoting collaboration among group members (Hmelo-Silver et al., 2007; Sockalingam, Rotgans, 
& Schmidt, 2011). It is noted in the literature that the role of the tutor or facilitator is critical to the successful 
implementation of PBL (Hmelo-Silver, 2004; Savery 2009). The tutor provides the initial guidance and 
supports with process skills, such as metacognitive modeling for individuals and groups, during collaborative 
group work (Savery, 2009). The tutor is responsible for both moving the students through the various stages 
of PBL and for monitoring the group process to assure that all students are actively involved (Barrows, 1988; 
Hmelo-Silver, 2004). 


2.1.1 Role and Types of Scaffolding or Guidance in PBL 


Scaffolding can be defined as support provided by a teacher, facilitator, tutor, peer, or a computer- or 
paper-based tool that allows students to meaningfully participate in and gain skill at a task that they would be 
unable to complete unaided (Belland, 2014). This concept of scaffolding has been connected to Vygotsky’s 
zone of proximal development (ZPD), defined as the “distance between the child’s actual developmental 
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level as determined by independent problem solving and the higher level of potential development as 
determined through problem solving under adult guidance and in collaboration with more capable peers” 
(Belland, 2014; Reiser, 2004; Vygotsky, 1978, p. 86). Enabling the learner to bridge this gap between the 
actual and the potential depends on the resources or the kinds of support provided (Puntambekar & Hiibscher, 
2005). 

The original notion of scaffolding assumed that a single more knowledgeable person, such as a parent or a 
teacher, would help an individual learner, providing him or her with exactly the help he/she needed to move 
forward (Puntambekar & Hiibscher, 2005). Strategies for scaffolding include (a) enlisting student interest, 
(b) controlling frustration, (c) providing feedback, (d) indicating important task/problem elements to 
consider, (e) modeling expert processes, and (f) questioning (Belland, 2014; Wood et al., 1976). 

However, the reality of modern classrooms and the emergence of computer technologies has broadened 
the definition of scaffolding, expanding the potential sources of scaffolding and how scaffolds are delivered 
to students. Thus, scaffolds can be defined as tools, strategies or guides that support students in gaining 
higher levels of understanding that would otherwise be beyond their reach (Brush & Saye, 2002; Hannafin, 
Land, & Oliver, 1999; Simons & Ertmer, 2005). Scaffolds may assume multiple forms depending on the 
learning environment, the content, the instructor, and the learners (Simons & Ertmer, 2005). Brush and Saye 
(2002) conceptualize two categories of scaffolds: soft and hard scaffolds (Brush & Saye, 2002; Saye 
& Brush, 2002). Soft scaffolds are dynamic situational aid provided by a teacher or peer. Soft scaffolding 
requires teachers to continuously diagnose the understandings of learners and provide timely support based 
on student responses (Brush & Saye, 2002; Saye & Brush, 2002). This type of support is generally provided 
“just in time,” where the teacher monitors the progress students are making while engaged in a learning 
activity and intervenes when support or guidance are needed. In contrast, hard scaffolds are static supports 
that can be anticipated and planned in advance based on typical student difficulties with a task (Brush 
& Saye, 2002; Saye & Brush, 2002). Such scaffolds can take the form of printed materials, such as 
worksheets, scripted cooperation and structured journals (Hmelo-Silver, 2004; Schmidt et al., 2011), or 
embedded within multimedia and hypermedia software to provide students with support while they are using 
the software (Brush & Saye, 2002). 


3. COMPUTER-MEDIATED SCAFFOLDING: A DESIGN FRAMEWORK 


Incombered by the review of the PBL literature, a conceptual framework was constructed to guide the design 
and development of problem-based, self-directed, computer-based modules to support scaffolding in an 
online graduate level course. As shown in Figure 1, the framework was designed to propose how the core 
characteristics of PBL can be used to create computer-based hard scaffolds to facilitate problem-based 
learning in the absence of an instructor or facilitator. 


Individual Activity Team Activity 


Hard Scaffolds Self-Study Hard Scaffolds Soft Scaffolds Hard Scaffolds 
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3.1 Top Hat: An Interactive Content Development Tool 


A web-based PBL environment was specifically designed and developed for the purposes of this study. Top 
Hat was selected and used to develop the instructional materials. Top Hat is a commercially available, 
web-based teaching platform which offers two products marketed towards engaging higher education 
students in and outside of class: “Lecture,” a classroom response system which allows for interactive slide 
presentations, and “JnteractiveText,” interactive learning materials to help students study. The latter product 
was used to develop the PBL modules for this study. 

Interactive text, is a modern conceptualization of traditional textbooks. One of the unique elements of 
InteractiveText is the ability to embed questions within content. There is also a feature to allow students to 
respond with a drawing or graphical representation. Additionally, students can view each other’s responses 
and engage in dialogue within the threaded discussion. These questioning functions allow an instructor to 
assess students’ understanding in real-time or in advance of a class session. An instructor could use the data 
from assigned InteractiveText to identify issues and direct instruction towards areas where students are 
struggling. 


3.2 Incorporating Scaffolds in Top Hat Modules 


Using the Top Hat environment and the proposed framework, three online modules were developed. Each 
module focused on a targeted content embedded in an ill-defined problem statement. The students were to 
carefully read and analyze the problem statement to identify underlying concepts and issues, study them 
individually and then meet with their collaborative group to discuss and come up with the best solution. 
Thus, each module began with a real-world, ill-defined problem, which was used as the context for the 
instruction. In order to replace the human tutor who would normally guide learners’ discussion when 
reviewing and analyzing the problem statement, a series of hard scaffolds in form of consecutive questioning 
were provided to assist students in problem identification and analysis while exploring and applying 
conceptual and domain-specific knowledge. Furthermore, the successive questioning as a hard scaffold was 
aimed to encourage deeper thinking, elaboration, and argumentation. The following explains two types of 
hard scaffolds that were used to assist the learners. 

Analytical questions. Following the presentation of the problem, a series of “thinking questions” were 
provided to assist the learner in analyzing the problem. This line of questioning was designed to act as 
cognitive and metacognitive scaffolds by modeling the types of questions students should be asking of 
themselves. The thinking questions were related to both the domain-specific thinking, as well as 
self-regulation skills. 

Domain-specific guiding prompts and questions. In addition to analytical questions, students were 
guided to review related readings and other multimedia materials to explore domain-specific knowledge. The 
resources provide real-world examples and explanations of theoretical concepts, model expert 
behavior/thinking, or demonstrate a concept in action. 


4. THE METHODOLOGY 


As indicated earlier, the Top Hat modules were designed to support and be incorporated into the activities of 
an existing online course. “Organization and Management of Instructional Technology” is an elective course 
offered to students in an instructional technology graduate program. Participants enrolled in this course are 
primarily graduate students seeking a master’s degree or a certificate in instructional technology. The study 
was piloted during the fall semester 2016 and was conducted again in the spring semester of 2018. During 
fall 2016, the course met regularly once a week for three hours via WebEx while during spring 2018 no 
weekly virtual meeting was scheduled and the course was delivered asynchronously. In fall 2016, participants 
were both on campus (1) and at a distance (4) logging in for the class through the WebEx teleconferencing 
system. The researchers were present in the classroom during class sessions to observe live discussions and 
take notes. 
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The following questions guided the data collection process: 
e To what extent do the Top Hat modules impact students’ content knowledge acquisition and 
thinking skills? 
e To what extent do the hard scaffolds in the Top Hat modules affect students’ thinking and 
argumentation skills? 
e What were students’ perceptions of the Top Hat modules? 

A design-based approach was used to systematically study the process of implementing and evaluating 
the learning materials. According to this iterative approach, the intervention could simultaneously be 
designed, developed, implemented and studied (Wang & Hannafin, 2005). During implementation, formative 
evaluation data was collected to systematically analyze the effectiveness of the modules and identify changes 
before implementing it again in spring 2018. This paper reports the results of 2016 pilot study. Both 
qualitative and quantitative data were collected and analyzed during fall 2016 to inform decisions. Data was 
collected from multiple sources including pre-posttests, student written responses to open-ended questions, 
student perception surveys and classroom observations. 

Participants. A group of 5 students (3 female, 2 male) volunteered to participate in the evaluating the 
Top Hat intervention in fall 2016. None of the volunteers had previously participated in or completed the 
course. Additionally, three of five participants were completely new or novice to the process PBL. 


5. RESULTS 


Pre-posttest results for three modules were analyzed. Tests contained both close-ended questions 
(i.e., multiple choice and True/False) questions and open-ended questions which required short written 
responses. Close-ended questions were scored for correctness, while open-ended responses were scored using 
the rubric described above. In general, participants achieved an increase in their overall score from the pretest 
to the posttest. The average score among participants demonstrated growth in content knowledge and 
achievement for all three modules (Table 1). 


Table 1. Summary of average pre-posttest growth 


Test Points Possible pverace, * Rates pveriee Fastest Average Growth 
Score Score 

Module 2 22 100% 13 59% 16.5 75% 3.5 16.5% 

Module 3 23 100% 14.6 63% 18.3 80% 3.7 16% 

Module 4 19 100% 11.5 62% 13.75 72% 2 11% 


While the results demonstrate that participants’ scores improved from pre to posttest, each module 
pre-posttest included items which resulted in an average decline or minimal gain in scores among participants 
(Table 2). 


Table 2. Summary of items with negative or minimal gain 


Average Pretest | Average Posttest « 
. : Scbre Seane verage Growth 
Test Item | Points Possible 
# % # % # % 
Module 2 3 4 2.5 63% 2.75 69% 0.25 6% 
6 4 2.5 63% 2.25 56% -0.25 -6% 
Module 3 1 5 4.6 92% 4.2 84% -0.40 -8% 
Module 4 2 3 1.25 42% 2 67% 0.75 25% 
4 5 3 60% 3.25 65% 0.25 5% 


Item 6 in the Module 2 experienced a 6% decline in average scores. The majority of participants’ scores 
did not change between pre and post, while one participant’s score declined. Additionally, Item 3 showed 
little gain (6%) compared to the other items. Each of these items asks participants to explain a concept and 
provide an example to support their answer. The lower scores reflect a difficulty providing reasoning and 
argumentation for more complex concepts within written responses. It is worth noting that Item 4 required 
participants to use the context of their course project in their response. While the item did achieve a small 
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gain (5%), it is telling that the module did not appear to improve their performance when applying the 
content to their project. 

To assess the effects of hard scaffold on students’ thinking and argumentation skills, written responses to 
open-ended assessment items within the Top Hat modules were scored and analyzed (see Table 3). The 
online activities for Modules 3 and 4 consisted of two parts; an individual activity followed by a team 
activity. The two parts were scored separately. 


Table 3. Summary of Top Hat scores 


. ; Average Score 
Top Hat Module Points Possible rr % 
Module 2 47 28.1 65% 
Module 3: Individual Activity 5 3.25 65% 
Module 3: Team Activity 21 12.25 58% 
Module 4: Individual Activity 8 4.67 58% 
Module 4: Team Activity 15 7 47% 


The average scores for open-ended responses were low (Table 4). However, it should be noted that 
among the five participants scores often varied widely per item. Scores appear to have been impacted by the 
weight placed on the level of argumentation and justification. Many open-ended questions consisted of both 
Level 1 and Level 2 components described in the scoring rubric (worth 4 points or more). The mediocre 
average scores reflect a general difficulty with the skills of argumentation and justification which made up 
the bulk of the possible points. 

Participants generally lacked detailed reasoning or justification to support and explain their thinking. This 
could be because students are not used to defending their thoughts in writing, lack skills and knowledge 
regarding argumentation and reasoning, or that they felt rushed when completing the activity so they didn’t 
take time to fully expand on their thoughts for each question. The low scores could also point to a lack in 
conceptual content knowledge or prior experiences which could have affected their ability to provide 
thoughtful and well-constructed arguments. Additionally, the low scores could indicate that the 
metacognitive and cognitive scaffolds did not support or provide enough guidance for students to fully 
achieve the learning task. 

The results for team activities mirror the findings for individual activities. Scores were affected by the 
team’s level of argumentation within responses. Both teams tended to provide superficial answers with very 
limited reasoning or justification within their responses. However, it should be noted that the question 
prompts in team activities focused on the application of domain-specific knowledge and the presentation of 
team generated solutions. The question prompts did not necessarily specifically request reasoning in writing, 
but practice in the domain would expect justification for solutions. Teams were observed discussing their 
answers and reasoning, but this collaboration was not reflected in the written responses. This could be due to 
the fact that one team member was acting as the scribe for the team and did not include all the conversation 
or thinking that led up to the compilation of the response. Or, it could be due to the format of the question 
prompts and embedded scaffolds. It is evident that the scaffolds did not provide enough guidance to teams to 
elicit the expected components of the written response. 

The implementation of the Module 4: Team Activity consisted of the three participants enrolled in the 
course. As such, the team activity was designed to allow the team to use the context of the course project 
during the team activity. The opportunity to work within the context of their project did not appear to 
improve open-ended responses. However, the team was observed skipping past questioning scaffolds and 
moving straight from question prompt to question prompt. It can be assumed that the embedded questioning 
scaffolds did not attract attention within the modules or were perceived to be extraneous. The format of the 
question prompts directed much more attention as they required action from participants. 

A survey was conducted to evaluate student perception of the Top Hat modules. The survey consisted of 
23 items and each item was accompanied by a 5-point Likert scale, with 1 denoting the most disagreeable 
and 5 denoting the most agreeable. The survey questions were categorized under the dimensions of PBL 
approach, scaffolds, learning evaluation, and the web-based platform (Top Hat). The results of students’ 
perceptions of Top Hat materials are shown in Tables 4-7. 

Results indicate that participants generally agreed that the PBL approach was helpful and effective for 
interacting with and learning the content, as shown in Table 4. However, questions regarding skills associated 
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with PBL, items 4-10, such as collaboration and communication demonstrate a slightly wider range of 
responses, as illustrated by the standard deviation calculations. This is important to note because these skills 
are typically scaffolded by a facilitator in PBL during collaborative group sessions. During the Top Hat 
intervention, early group discussion on problem analysis was not facilitated by the instructor/tutor. The 
results demonstrate that the hard scaffolds were not sufficient to take the place of the presence of a facilitator, 
as scaffolds offered by an instructor or a trained tutor are provided on spot and in response to learners’ 
thoughts (soft scaffold). Thus, it is likely that a human facilitator could better model thinking processes and 
promote skills of communication, collaboration and critical thinking, especially with students that are new or 
novice to the PBL approach. However, with such a small sample size it is difficult to generalize. 


Table 4. Student perceptions of PBL approach 


No | Statement Mean | Std 


N 
1 The Top Hat modules helped me identify what I needed to learn more about. | 5 | 3.8 0.84 
2 The cases presented in the Top Hat modules were relevant. 5 | 4.4 0.89 
3 I used prior knowledge and experiences to help me analyze the cases. 5 | 4.6 0.55 
4 I had a chance to collaborate with other students. 5 | 4 1.73 
5 Interacting with other students improved my learning. 5 | 4.2 1.10 
6 I experienced quality interactions with the other students in terms of learning. | 5 | 3.8 1.79 
7 Learning by interacting with other students enhanced my confidence 5 | 3.8 1.79 
8 The interactions with the other students enhanced my communication skills. | 5 | 3.8 1.79 
9 The interactions with the other students enhanced my collaboration skills. 5 | 3.8 1.79 
10 | Working with group members helped me make connections between ideas. 5 | 3.4 1.67 


Students’ perception of the Top Hat scaffolding questions indicate that participants perceived the 
recommended resources embedded in the Top Hat modules to be helpful and effective for interacting with 
and learning the content. 


Table 5. Students’ Perception of Top Hat Scaffolds 


No | Statement N | Mean | Std 

1 The questions in Top Hat helped me identify critical issues in the cases. 5 | 3.4 1.67 
The questions in the modules prompted me to think more deeply. 5 | 3.2 1.48 

3 It was easier to learn with the guidance of questions incorporated in the Top Hat 5 | 32 1.48 
materials. , , 
The videos, articles and other resources included in the Top Hat modules helped 

4 5 | 44 0.89 
me make sense of the content. 

5 The Top Hat materials provided guidance to the construction of new knowledge. 5 | 3.8 0.84 


Table 6 shows how the participants evaluated the learning processes that they experienced. The results 
reveal that participants perceived the Top Hat product to be a helpful learning tool and the intervention to be 
effective. 


Table 6. Students’ evaluation of learning 


No | Statement N | Mean | Std 

1 Top Hat materials helped in my learning of the content. 5 | 3.4 1.14 

2 The Top Hat modules improved my understanding of the content. 5 | 3.8 1.10 

3 I have gained new knowledge as a result of completing the Top Hat modules. 5 | 3.8 1.10 
I feel better prepared to apply the content to my project after completing the Top 

4 5 | 3.8 1.10 
Hat modules. 

5 This type of activity is suitable for how I learn. 5 | 3.6 1.14 

Table 7. Students’ perceptions of Top Hat 

No | Statement N_ | Mean Std 

1 The Top Hat platform was easy to use. 5 4.4 0.89 

2 I enjoyed the Top Hat modules. 5 3.8 1.10 

3 I would use Top Hat again if given the opportunity. 5 4.4 0.89 
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6. DISCUSSION 


In general, the results of the 2016 study are encouraging for the use of hard scaffolds to promote 
problem-solving in an online environment, where the presence of a tutor is less feasible compared to a 
synchronous (virtual) or face-to-face learning environment. It appeared that hard scaffolds provided an 
opportunity for students to question their knowledge and thinking, and encourage them to dig deeper into the 
content. Also, the Top Hat modules and the embedded hard scaffolds are perhaps more important for 
asynchronous online PBL courses where regulating one's learning process is more critical. Furthermore, 
while participants’ reasoning skills did not seem to improve as a result of embedded hard scaffolds, their 
domain-specific knowledge seemed to show improvement, which indicates that participants gained 
conceptual knowledge. Additionally, participants seemed to find the learning materials engaging and helpful 
in learning the content and appeared to enjoy interacting with the course content through the Top Hat PBL 
modules. However, it is evident that the design and implementation of online modules could be used to 
modify the design of hard scaffolds embedded in the online PBL modules in spring of 2018. 
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